Comparison of Pixel- and Object-based Sampling Strategies for Thematic Accuracy Assessment

Confusion matrix and derived global indices (kappa, overall accuracy, producer accuracy) are widely accepted as a standard method for the accuracy assessment of land use/land cover maps. In order to build the confusion matrix, the ground truth labels of samples are crossed with the map labels. Most of the time, the sampling strategies are simply based on the spatial distribution of sample points (systematic, random or stratified random sampling). They do therefore not account for existing objects in the map. In this study, an object-based sampling strategy is compared with classical (pixel-based) sampling. Instead of selecting pixels on the basis of their coordinates, a random sampling was performed in the list of objects and the sampling probability was proportional to the object size. The central point of selected objects was used to create the confusion matrix, which differs from the standard confusion matrix by the fact that the weight of each object was on average proportional to its area.The performance of the two sampling strategies was quantified thanks to two sets of simulation : map alterations and sampling repetitions. An existing land cover map was regarded as ground truth and was spatially and/or thematically altered. Errors in the test maps were generated by blundering boundaries and/or changing object labels, respectively. The bias (difference between average estimate and the truth) and the variance of the overall accuracy estimates were then measured as an indicator of the robustness of the confusion matrix. Pixeland object-based sampling did not lead to the same measure of accuracy. The former evaluated the global accuracy (influenced by boundary errors) while the latter measured the thematic accuracy (only influenced by labeling errors). The other big difference between the two sampling strategies was a smaller confidence interval on the accuracy estimates with the object-based strategy. For a given confidence interval, object-based strategy could thus reduce the sampling effort. The proposed object-based sampling strategy was easy to implement and could help to reduce the costs of map validation. Further work is needed to determine a priori the number of object samples necessary to fulfill a given level of confidence.